Notes on Stein-Sahi representations and some 
problems of non harmonic analysis 

Neretin Yu.A. 

We discuss one natural class of kernels on pseudo-Riemannian symmetric spaces. 

Recently, Oshima |H7] published his formula for c-function for on pseudo- 
Riemannian symmetric spaces (see also works of Delorm ^] and van den Ban- 
Schlichtkrull 0, 0]). After this, there arises a natural question about other 
solvable problems of non-commutative harmonic analysis. 

In the Appendix to the paper |S7], the author proposed a series of non L^- 
inner products in spaces of functions on pseudo-Riemannian symmetric spaces 
and conjectured that this object is reasonable and admits an explicit harmonic 
analysis. 

In this work, we discuus the problem in more details, in particular, we obtain 
the Plancherel formula for these kernel for Riemannian symmetric spaces U(n), 
U(n)/0(n), U(2n)/Sp(n). We also give a new proof of Sahi's results |75] . 

0. Introduction 

0.1. Inner products defined by kernels. Starting the famous works 
of Bargmann fSJ and Gelfand-Naimark 21 , various inner products having the 
form 



(/i,/2)=// K{x,y)h{x)h{y)dxdy (0.1) 

J JG/Hy.G/H 

are quite usual in the theory of unitary representations. Here G / H is a homo- 
geneous space and K{x,y) is a distribution (a 'kernel') on G/H x G/H. The 
group G acts in a space of functions on G/H by transformations having the 
form 

p{9)f{x)^ f{gx)^{g,x), (0.2) 

where j{g,x) is some function ('multiplier') on G x G/H. We intend to obtain 
a unitary irreducible representation; under this requirement, the kernel K{x,y) 
is uniquely determined by the explicit expression for the multiplier 7(3, x). An 
actual evaluation of the kernel is not difficult. 

But the scalar square (/, /) of a function /, i.e., the integral 



(/,/)= // K{x,y)f{x)f{y)dxdy 

J Jg/HxG/H 

has no visible reasons to be positive. Usually, positive definiteness of a given 
inner product of the form (|0.1|l is a nontrivial problem. 

Example. We consider group SU(1, 1) consisting of 2 
times2 matrices ( ^ ^ ) , where |a^| — |5p = 1. It acts in the space of functions 



a 
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on the circle |z| = 1 (or z — e'P'*') by the formula 
These operators preserve the inner product given by 



(/,5)= JJ\zi-Z2\-'-'hiz,)f2{z2)dvidip2 (0.4) 

This inner product is positive definite iff — 1 < s < 1. 

The inner product HU.4|I is a Sobolev inner product. Also, the Sobolev inner 
products in spaces of functions on spheres appears in the representation theory 
of S0(1, n). Some anisotropic Sobolev spaces arise in the representation theory 
of rank 1 groups. But usually inner products define functional spaces that 
are uknown in the analysis. The spaces discussed in this paper can be concidered 
as some kind of " Sobolev spaces of matrix variables. 

0.2. Stein— Sahi complementary series. In 1967, E. Stein [HI] con- 
structed an extremely degenerated complementary series of unitary represen- 
tations of GL(2n, C). Recall his construction. Consider the space Mat„(C) 
consisting of complex n x n matrices. Consider the group GL(2n,C) consisting 

of (n + n) X {n + n) invertible complex matrices g = \ K\ ■ This group acts 



on Mat„(C) by linear- fractional transformations 

z ^ {a + zcy^{b + zd). (0.5) 

Fix (7 G M Consider the action of GL(2n, C) in the space of functions on Mat„(C) 
by the operators 

Pa{9) f{z) = f{{a + zc)-\b + zd))\detia + zc)!-^"— . 

Define the following Hermitian form in the space D(Mat„(C)) of smooth func- 
tions on Mat„(C) with compact support 



\detix-y)\''"+^f{x)giy)dxdy. (0.6) 

J JMat„(C)xMat„(C) 

For a > 2n this integral converges, further we consider its meromorphic contin- 
uation in a to the whole plane ct G C. 
Stein proved that for — 1 < ct < 1, 

1. the Hermitian form (•, is positive definite; denote by Dia the completion 
of 'D(Mat„(C)) with respect to (■, ■)^ 

2. the operators Pa{g) are unitary in "K^. 

The Stein- type constructions exist for all the series of classical groups, pre- 
cisely for the groups 

GL(2n, C), GL(27i, R), GL(27i, M), (0.7) 
0(2n, 2n), U(n, n), Sp(n, n), (0.8) 
SO(47i,C), Sp(27i,C), S0*(4n,C), Sp(2n,M). (0.9) 
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The cases HU.7|I were considered in the classification work of Vogan C|57). 
Section 2), see also |7Sj. Sahi [7H]-|77] constructed analogs of the Stein repre- 
sentations in the remaining cases. 

0.3. Berezin kernels and problem of existence of non-L^-theories. 

The explicit Plancherel formula for on Riemannian noncompact symmetric 
spaces was obtained by Gindikin and Karpelevich in 1964. In 1978, Berezin 
[7] ^ observed that the space on a classical Hermitian symmetric space admits 
a natural deformation. 

For dcfiniteness, consider the Hermitian symmetric space 

G/K^\J{p,q)/\]{p)x\]{q), p^q 

Following E.Cartan, we realize this space as the space "Bp^g oi p x q complex 
matrices with norm < 1, the group \J{p,q) acts on this space by the linear- 
fractional transformations z ^ {a + zc)^^ {b + zd) . Consider (see |S7] for details) 
the space 1){'Bp q) of smooth function on "Bp^g with compact support. Denote 
the representation of the group U(p, q) in this space by the shifts 

j)^W = ^((« + ^c)-i(6 + zrf)). (0.10) 

For cr S M, define the inner product in ©(CBp ,j) by 

dct(l - zz*y det(l - uu*y 
I det(l - zu*)\ 



{Fi,F,),= II ' ' ^-F,{z)F,iu)da{z)da{u), 



where 



(0.11) 

da{z) = dct(l - zz*)-P-'^ Yl {^dzk,i dzki) 



is the U(p, (7)-invariant measure on Bp g. A simple calculation shows that the 
form HO.llfl is U(p, (7)-invariant.^ 
It turns out to be that for 

(7 = 0, 1, . . . ,p — 1, or a > p — 1, 

our inner product is non-negative definite. 

The latter statement is a reformulation of the well-known theorem (Berezin- 
Gindikin-Rossi-Vergne-Wallach) on unitarizability of scalar highest weight rep- 
resentations (0, Unii see a relatively recent exposition in jl8j). 

^The first work that can be attributed to this subject, is Vershik, Gelfand, Graev, |84| . I20| . 
These authors apply the representations pu of the groups U(l,g) to construct representations 
of current groups; for a collection of other early references, see I5bl . 

^Compare the Stein inner product IU.6I and the Berezin inner product lU.lli . In the first 
case, the kernel has a singularity on the surface det(x — y) = 0. This surface itself is singular, 
and its most singular strate is the diagonal x = y. On the contary, the Berezin kernel has no 
singularities in 25p,q X "Bp^q at all. We can substitute to lU.lli arbitrary compactly supported 
distributions unstead of functions _Fi , F2. 
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Consider the completion IK^ of the space D(Sp,q) with respect to the inner 
product (|0.11l) . and the unitary representation (|0.10|l of U(p, q) in this space. 

At first sight, the representation po- given by the formula (|0.1l)|) does not in- 
clude a. Really, for sufficiently large a, all the representations are equivalent 
to the standard representation of U(p, q) in L^(U(p, q)/\J{p) x U(g). Moreover, 
the natural hmit of the spaces as ct — > oo is [U(p, q)/'U{p) x Viq)] 

But the Plancherel formula for pa- depend on a. For sufficiently small cr > 0, 
the spectrum of pa undergo several bifurcations.^ It is possible to obtain an 
explicit decomposition of the Hilbert space JCo- into pieces with uniform spectra, 
see 

Next, Pa admits a natural analytic continuation to negative integer a, and 
the corresponding hmit as cr — > —oo is [U(p + q)/V{p) x U(g)] . Thus, we 
obtain some kind of interpolation between 

[U(p, q)/\J{p) X U(q)] and [V{p + q)/V{p) x U(<z)] 

In his short note iTj, 1978, Berezin gave (without proofs) the explicit Plancherel 
formula for representations pa (in the case of Hermitian symmetric spaces) for 
sufficiently large a (before the start of bifurcations). Berezin died soon after 
this, and the first known proof of his theorem was published by Unterberger 
and Upmeier |H21, 1994. 

In preprints [S^, [SB], 1999, the author concidered the Berezin represen- 
tations for arbitrary classical Riemannian symmetric spaces, and obtained the 
explicit Plancherel formula for arbitary a. ^ ^. 

Some steps in analysis after the Plancherel formula were undertaked in |S7] , 
|59|. |58|. Also in [^0] it was obtained a p-adic analog of Berezin kernels.^ 

Since there exists a reach non-L^-analysis on Riemannian symmetric space, 
there arises the following question: 

— are there other homogeneous spaces G/H and other kernels K{x,y) ad- 
mitting an explicit and interesting harmonic analysis? 

Since we discuss the harmonic analysis, a representation p having the form 
H(J.2|I must be reducible, and hence the distribution K{x, y) in the formula (|0.1|l 
is not uniquely defined. At a first sight, a choice is too large. 

Indeed, for each function K{z, u) on 23p^g x 'Bp ^ satisfying 

L[{a + zcy^{b + zd), {a + ucy^{b + ud)] = L[z,u], (0.12) 

^This representation is equivalent to the tensor product of a scalar highest weight represen- 
tation of U(p, q) and the complex conjugate representation; it is natural to consider a similar 
and more general problem about the tensor product of a highest weight and lowest weight 
representation, in this case, the Plancherel formula was obtained by Zhang, |88| . 

*In the work of Olshanski and the author 1641 (see also 1651 . 1501 V this phenomenon was 
applied to construction of exotic unitary representations of 0{p,q) and U(p, 5). 

^ Some modifications of proofs of results of 1541 were obtained later in 1151 and 1891 , see also 

ESI, EH 

^Apparently the Plancherel formula for all the groups can be reduced to some single identity 
with the Heckman-Opdam spherical hypergeometric transform (see, for instance, 1281 '). as far 
as I know, this possibility is yet not realized. Apparently also, that the Plancherel formula 
for Berezin kernels can be interpolated as an exotic Plancherel formula for Heckman— Opdam 
spherical transform (this is correct in rank 1 case, see 1581 . 
Only for Bruhat-Tits buildings of the series Ari- 
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the expression 




L{z, u)Fi{z) F2{u) dcr{z) da{u) (0.13) 



is invariant with respect to the translation operators 

F{z) ^ F{{a + zcy\b + zd)). 

A kernel L satisfying (|0.12() can be represented in the form 

L{z,u) = ^(Ai, . . . , Ap), 

where Aj are the eigenvalues of the matrix 

(1 - zz*)-\l - - uu*)-\l - uz*) 

and £ is a symmetric function on the octant Xj > 0. There are many such 
functions, and hence there are too much invariant kernels L. 

Nevertheless, the existence of explicit harmonic analysis is a non-formal, but 
very strong restriction; some experience in the analysis on rank 1 symmetric 
spaces (or reading the book [ZO]) shows that a collection of possibilities is not 
large^; for spaces of rank > 1 situation became strongly rigid and even an 
existence of examples is not obvious. 

0.4. Natural kernels on pseudo-Riemannian symmetric spaces. In 

the first approximation, the problem that we formulate is a problem of restric- 
tion of Stein-Sahi representations to some special symmetric subgroups. 

More precisely, for any classical symmetric space G/H there exists an over- 
group G D G acting locally on G/H {^). If G is in the list JO.VIIO.Hll (and 
its action on G/H is locally equivalent to a fractional-linear action on a matrix 
space M, see below Section 4), then our problem is a problem of restriction of 
a Stein representation of G to the subgroup G. In fact, we can forget about 
the group G and consider only the restriction K{-,-) of the Stein kernel to the 
symmetric space G/H and action of G in the Hilbert space defined by this ker- 
nel. For some value of the parameter s, we obtain the natural action of G in 
L^iG/H). 

It is possible to define the kernel K{-, •) in the terms of the symmetric spaces 
G/H themselves; this allows to extend our problem to the case, then G is not in 
the list HU.7I) -- (|U.9I) . But existence of an island of positivity of the form in these 
cases becomes a problem. 

0.5. Structure of the paper. In Section 1, we dicuss our kernels for the 
space G/H ~ U(n) x U(n)/U(n). This also gives a realization of the Stein-Sahi 
representations of U(n, n) (For this case, more detailed discussion including the 
unipotent representations is contained in [fi2J). 

^There is lot of papers of Molchanov on this subject, for instance 1391 . 1411 . 1141 
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In Section 2, we repeat the construction for spaces U(n)/0(n), U(2n)/Sp(n) 
and for their overgroups Sp(2n, E), S0*(4n). 

The proof of positivity is contained Section 3, it is based on the exphcit 
expansions of the kernels in spherical functions. This done uniformly using 
Kadell's generalized Selberg integral. 

In Section 4, we briefly discuss the remaining series of classical groups. 

In Section 5, we formulate in details the problem discussed above in 0.5. We 
also try to explain why the problem looks as solvable. 

0.6. Acknowledgements. I thank V.F.Molchanov, G.I.Olshanski, T.Koba- 
yashi, E.van den Ban, and T.Oshima for discussion of the subject. 

This work was done during my visit to the Erwin Shrodinger Instirute, Vi- 
enna, in winter 2001-2002. I thank the administrators of the Institute for hos- 
pitality. 

0.7. Notations. 

Let A — {ttij} be a matrix. Then A* is the transposed matrix, and A* is 
the adjoint matrix, i.e., matrix with elements aji. Also, we denote by A the 
element- wise conjugate matrix, i.e., the matrix with the matrix elements a^-. 

A matrix x over R, C is symmetric ii x = and skew- symmetric if a; = —x. 

A matrix over R, C, or quaternions H is Hermitian ii x = x* and anti- 
Hermitian if a; = —x*. 

If A is a Hermitian matrix, the notation A > means that A is positive 
definite, i.e., hAh* > for each vector-row h. 

The symbol ||z|| denotes the norm of an operator z in a Euclidean space. 
The value ||zp coincides with the maximal eigenvalue of z*z. 

We use the standard notation for the Pochhammer symbol 
{a)k := a{a + 1) . . . (a + fc — 1) 

For a complex number x we use the following notation for its powers 

The symbol [a] denotes the integer part of a. 

1. Sobolev kernels on the spaces U(n) 
and unitary representations of U(n, n). 

On construction of this section, sec also I(i2| . 

1.1. The group U(n). The unitary group U(n) is a compact Riemannian 
symmetric space 

GjK ^ U(7i) ~ U(n) X U(n)/U(7i). 

Indeed, the group G := U(n) x U(n) acts on U(n) by left and right multiplica- 
tions 

(ft,i,/i2) : z i-^ hi^zh2. 
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The stabilizer K ~ U(n) of the point z = 1 consists of elements {h, h) € 
U(n) X U(n). Using transformations z i-^ h~^zh, each element of U(n) can 
be reduced to the diagonal form 



/e*^i ... \ 
e'*'^ ... 



\ ... e'^"/ 



(1.1) 



1.2. Overgroup. The space G/K admits a larger group of symmetries, 
namely the pseudounitary group U(n,n). Recall that U(n,n) is the group of 

(n + n) X (n + n) matrices g = ^ ) preserving Hermitian form H{-, •) with 



1 

the matrix [ ^ i.e., 



il(w © v' ® w') := ^ UjtJ^- — ^ WjW^-- 

In other words, the matrix g satisfies the condition 

a bV fl \ fa b\ ^fl 
c di lo -l) [c d) ~ [o -1 



(1.2) 



The group U(n, n) acts on the space U(n) by linear fractional transforma- 
tions 

z<-^ z^s^ := ia + zc)-^{b + zd). (1.3) 
Lemma 1.1 Let z € U(n), g S U(ri, n). 

1.3. Proof of Lemma 1.1. Identification of U(n) with a Grassman- 

nian. For z G U(n) consider its graph C C"©C". A vector w©w e C"©C" 
is an element of if w = vz. 
For v,v' ^ C", we have 

H{v ® vz, v' © v'z) = {v, v')cr^ - {vz, v'z)c^ = {v, u')c" - {v, v')c^ = 0, 

where (•, denotes the standard inner product in C". Hence 14 is a maximal 
if-isotropic subspace^ in C" © C". 

Conversely, let F be a maximal -ff-isotropic subspace in C" © C". Since the 
form H is strictly negative on OffiC", we have Fn(OffiC") = 0. But dim V = n, 
and hence y is a graph of some linear operator z : C" © ^ © C". By the 
isotropy condition, 

= H{v © vz, v' © v'z) = H{v © 0, -u' © 0) - H{0 ®vz,Q® v'z) 



^Consider a linear space L equipped with bilinear (sesquilinear) form B{-, ■). A subspace 
M C -L is isotropic if B{h, h') = for all h, h' 6 M. 
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and hence z G U(n). 

Thus we obtain the natural identification 

The Grassmannian of maximal 1 J The space U(n) = 

i/-isotropic subspaces in C" ©C"J ' ' [ = U(n) x U(n)/U(n) 

The group U(n, ri) preserves the Hermitian form •) and hence it transfer 
isotropic subspaces to isotropic subspaces. Thus U(n, n) acts in a natural way 
on U(n). It remains to write an explicit formula for this action. 

Lemma 1.2 We have gVz — Vu, where u — z^^^ [a + zc)^^(6 + zd). 
Proof. We have v ® vz &Vz. Applying g, we obtain 

v{a + zc) ® v{c + zd) G gVz- 

Denoting w — v{a + zc), we obtain the required statement. □ 
1.4. Jacobians. 

Lemma 1.3 Denote by ^i{z) the Haar measure on the group U(n), denote by 
^(zlsl) its image under the transformation z ^ z^^\ given by Then 

/^(zlsl) = |det(a + zc)r2"^(5) (1.4) 



Proof. Let h e U(n) x U(n) C U(n, n) be an element of the form 

Then z^''! — u^^zv. The Haar measure is invariant with respect to such trans- 
formations. 
Denote 

7(5,^) := |det(a + zc)|-2". (1.6) 
This expression satisfies the identity 

for hi, h2 having the form H1.5|l . Due the invariance of the Haar measure, the 
Jacobian (or Radon-Nykodim derivative) 

A.(zM)/m(^) 

satisfies the same condition. 

Hence, without loss of generality, we can assume that z = 1, z^9] = 1. The 
tangent space to U(7i) at this point consists of matrices A satisfying A + A* = 1. 
We denote this space of all the anti-Hermitian matrices by AHerm. 
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It is easy to show (see, for instance, Lemma 1.1), that the differential 
of a linear-fractional transformation z i— > z^^l is given by 

A ^ (a + zc)^ iA(-cz[9l (1.7) 

this formula is valid on whole space of n x n matrices for g £ GL(2n,C), in 
particular it can be used in our case. 

Now z — z^^^ = 1 and hence a + c ^ b + d. This equation, together with 
p. 211 allows to transform H1.7|) to the form 

{a + c)-^A[{a + c)-^]*. (1.8) 

It remains to evaluate the determinant 5{P) of the linear transformation 
AHerm(n) —^ AHerm(n) given by 

A^PAP*, PeGL(n,C). 

Obviously, S{P) is a homomorphism from GL(n,C) to the multiplicative group 
of positive numbers. Hence 6{P) = 1 on U(rt). Each P G GL(?i,C) can be 
represented in the form uAv, where u, v E U(n), and A is a diagonal matrix. 
Obviously S{A) = |det(A)|2" and hence S{P) = |det(P)|2". This finishes the 
proof. □ 

1.5. Invariant kernels on U(n). Denote by 'B„ the set of n x n matrices 
z with norm < 1; an equivalent condition is 1 — z*z > By S„ denote the set 
of all n X n matrices with norm 1; an equivalent condition is 1 — z*z ^ 0. 

Fix cr G C. For z G "Bn, we define 

(1 _ :^ ^ L^,., (1.9) 

the series in the right-hand side is convergent. 

Fix (T, T G C. We define the hmction det(l — z)^"'^^^ depending in the 
variable z G by 

det(l - z)^'"l^> = det [(1 - zy] det [(1 - z)^j . (1.10) 

This expression is a smooth C- valued function on 13„. For z G 'B,i, we define 

det(l - z){''l^> lim det(l-(l-e)z){'"l^}. (1.11) 

This limit exists outside the surface det(l — z) — 0, and the expression l|l.ll|l is 
a smooth function on the set 

1 - zz* > 0, det(l -z)^0. 

Obviously, for a unitary matrix h G U(n), 

det(l - /iz/i-i){'"l^> = det(l - z){'"l^>. 
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Lemma 1.4 For z E U(n), denote by e*^^,. . . , e*'^"- its eigenvalues. Then 

n 71 

det(l - z){-l-> = exp{(a - r) J^i^k ~ 7r)/2} • J] | sin(i^fe/2)r+^ 

fc=i fc=i 

This statement is more-or-less obvious, for a formal proof, see \ <62\ 
We define the kernel L„^r{z,u) on U(7i) by 

= det(l - 2,ueU(n). (1.12) 

Lemma 1.5 a) For r, cr e M, t/ie kernel L„^t{z,u) is Hermitian, i.e., 

L„^r{u, z) = Lcr,T{z, u) . 

b) The kernel Lcr.T{z,u) is \J{n) -invariant, i.e., 

La,Tihizh2, hiuh2) = L^^riz, u), hi, e U(ri) 
The both statements are obvious. 

We define the sesquilinear form /^^r on C°°((U(n)) given by 

IaAFi.F2)^ [[ L„Az,u)Fliz)F^)d^i{z)d^l{u) (1.13) 

If (7, r G M, then the form is Hermitian. The integral is convergent if Re(cr + 
r) > —1. By general reasons, the integral admits a meromorphic continuation 
to the whole plane {a, r) e (see [H], |2]). In fact, our proof of positivity given 
below is based on an the explicit construction of this meromorphic continuation. 

1.6. Action of U(rt, n). 
Lemma 1.6 The operators p^.r given by 

F{z) = F[(a + zcy\b + zd)] det(a + zc)^~"-'"l^"-^> (1.14) 

preserve the form I^^ j- 

Before starting a proof, we give some preliminary comments on formula 
(dHI). 

1. We must define these operators more carefully. First, 
det(a + zc){-"ll-"} = | dct(a + zc)|-2" 
and this expression is well-defined. Further 

det(a + zc){-''ll-^> = det(a)^-'"'l-^} dct(l + zca-^)^-"^^-^^ (1.15) 



10 



Since ||ca ^|| < 1 (this easily follows from (|1.2|l '). we have also \\zca ^|1 < 1. 
Thus the factor det(l + zca-i){-'^ll-^> is well defined, see (dUl, ifTTUIl . It 
remains to define 

det(a)^"'"ll"^> = I det(a)r'""^ exp{i(r - a) arg(a) + 2m{T ~ a)k], 

where k ranges in Z. If (cr — r) e Z, then the last expression is well defined, and 
Pa^T is a linear representation of U(p, q). 

Otherwise, the multi-valued function H1.15|l on 23„ splits into a countable 
family of smooth branches^°; these branches differs by constant factors exp{27rj(T— 
(T)fc}. Thus the formula (|1.14|) for a given 5, defines a countable family of opera- 
tors Pcr^r, which differ one from another by constant factors. We can choose one 
such operator in an arbitrary way, and then we will obtain a unitary projective 
(see, for instance, 35 , Section 14) representation of U(n,n), 

Pa.r{g)Pa.r{h) ^ >c{g,h)p„^r{9h), x{g,h)eC*. (1.16) 

2. Since dct(a) does not vanish, the function Indet(a) is a well-defined 
function on the universal covering U(n,rt)~ of \J{n,n). Hence the expression 
l|1.15|l is a well defined single- valued expression on U(7i, rt)~; so we can consider 
Po-.r as a linear representation of U(n, n)~, 

Pa.,r{g)Pa.,T{h) ^ Pa,T{gh). (1.17) 

3. We must prove the identities H1.16|l . (|1.17|l . A direct calculation is not 
difficult, but it is more reasonable to avoid it. Denote 

v{g, z) := det(a -f zc)^-"-'"l-"-^>. 

The desired identities are reduced to the "cocycle identity" 

v{gh,z)^y{Kz)v{g,z^'^'^). (1.18) 

But i^(g, K) is a power of the expression ^{g, z) given by (|1.6() . the latter expres- 
sion is the Jacobian. For a Jacobian, the cocycle identity is obvious. 

4. Let r = 0. Then our construction gives a representation of the Harish- 
Chandra holomorphic series. The kernel Lcrfi is the standard reproducing kernel 
for these representations, see, for instance, [S], UHli (H^- Representations of 
holomorphic series admit realizations (see |64p in spaces of holomorphic func- 
tions, in spaces of distributions on matrix balls 23^, and in spaces of distributions 
on Shilov boundary V>(n) of the matrix ball. The last variant corresponds to 
our realization. 

For cr = 0, we obtain a lower weight representation. 

5. Our construction is invariant with respect to the shift 

(cr,r) (cr + l,T-l). 
^"in particular, our operators preserve the space C°°(U(n).) 
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We will not refer to this remark below, for discussion, see jHS, 2.9. 

1.7. Proof of Lemma 1.6. We use the following simple identity 

det(l - zlffl [ulsl] *) = det(l - zu*) det(a + zc)-^det{a + ucf^ 

valid for g ~ ^) ^ Keeping it in mind and using the formula (|1.4|l 

for the Jacobian, we substitute z ^ z^^l, u ^ u'^l to integral and obtain 

Lcr,Tiz,u) det{a + zc)^^'^^^'^^det{a + uc)^ ' '^x 

U(n)xU(n) 

X Fi(zM)F2(ulsl)det(a + zc)"^" det(a + uc)-^" dfi{u) 

Q.E.D. 

1.8. Unitary representations of U(n, n). 

Theorem 1.7 Let a, t ^ Z,. The Hermitian form I„ r given by ^.1S\I is definite 

hr] = [c7 + n]. 

Proof is given below in Section 3. More elementary proof is contained in 
|62|. This paper also contains a picture of a domain of positivity. 

It is convenient to introduce new parameters t, s by 

a = -n/2 + s, T = -n/2 + t. (1.19) 

In this notation, the cases of even and odd n are slightly different. 

a) For an odd n the condition of positivity is 

|s-j|<l/2, \t-j\< 1/2, for some jeZ. 

b) For an even j the condition of positivity is 

^G[j>i + 1]: se[j~l,j] for some jeZ. 

Proposition 1.8 // s = t, then the Hermitian form I^^r coincides with the 
L'^ -inner product 

{Fi,F) = / F,{z)F^dfi{z). 

Jv{n) 

See a proof in 62 , this also can be easily derived from calculations of our 
Section 3. 

Under the conditions of Theorem 1.7, we denote by 0-Ca,T the completion of 
C°°(U(n)) with respect to this form. We obtain that our representation po-.r in 
this case is unitary in H„ t. 

-'-'^ recall that {x\ denotes the integral part of x. 
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2. Sobolev kernels on the spaces l]{n)/0{n), U(2n)/Sp(n) 
and unitary representations of the groups Sp(2n, M), S0*(4n) 

Here we show that the space U(n)/0(n) can be reahzed as the space of 
unitary symmetric matrices and the space U(2n)/Sp(n) as the space of unitary 
skew-symmetric 2n x 2n-matrices We show that the groups Sp(2rt,R) and 
S0*(4n) respectively act on these spaces by hnear-fractional transformations. 
Then we restrict the kernel L„ t defined by H1.12() and obtain unitary represen- 
tations of Sp(2n,M) and S0*(4n). 

A. Spaces U(n)/0(n). 

2.1. Symmetric spaces U(n)/0(ri). Now let X„ be the space n x n 
matrices z satisfying the conditions 

ZZ* = 1, Z = 0*. 

The group U(n) acts on X„ by the transformations 

h : z I— > h^zh. 

The stabilizer of the point z = 1 is the real orthogonal group 0(n). It can be 
easily verified that action of U(n) on X„ is transitive. Thus, 

X„ ~U(n)/0(n). 

Each element of X„ can be reduced by the transformations z i 
h G 0(n), to the form 



fe'f' ... \ 
e'f'^ ... 



V ... e*v'"/ 



(2.1) 



The collection ipj is uniquely defined up to permutations. 

A matrix A is an element of the tangent space to X„ at z = 1 iff 

A = A*, and iA is a real matrix. 

2.2. Overgroup. Again, our space X„ admits a larger group of symmetries, 
namely Sp(2n,R). 

To observe this, we realize the real symplectic group Sp(2n,R) as a group of 
complex (n + n) x (n + n) matrices g preserving the Hermitian form in C" © C" 

with the matrix and the skew-symmetric bilinear form B having the 

matrix I • other words, 

1 0\ /l 0\ t f A_/^0 1 



9 -1 ■'^= -1 ' 3 -1 ■'^=V-i 



^^A remark for experts. These spaces are realized as the Shilov boundaries of the bounded 
Cartan domains Sp(2n, 1R)/U(n), SO* {4n)/U(2n), see [TI]. 
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These two identities also imply that the matrix g has the following block struc- 
ture 

Equivalently g preserves the real subspace of C" ® C" consisting of vectors h(Bh. 
The group Sp(2n,IR) acts on X„ by linear fractional transformations 

g : z ^ z'fl ($ + + z$). 

Lemma 2.1 Let z e X„, 5 e Sp(2n,M). Then e X„. 

Proof. As above, for a matrix z G X„, we consider its graph C C" ©C". 
Literally repeating the considerations of Subsection 1.2, we obtain that the 
subspace Vz is maximal isotropic with respect to the Hermitian form H. 

The condition z = z* means that Vz is Lagrangian (=maximal isotropic) 
with respect to the skew-symmetric form B. Indeed, 

B{v © vz, w ® wz) — vzw — wzv 

where v, w are matrices-rows. Since z = z*, we obtain zero in the right-hand 
side. 

Thus, we obtain the identification 

Grassmannian of n-dimensional subspaces in C" C" 
isotropic with respect to the both forms H and B 

{The space X„ — U(n)/0(n) of n x n matrices] 
z satisfying z — z* , z* z — 1 \ 

An element g G Sp(2n,R) preserves the both forms H, B and hence it maps 
our isotropic Grassmannian to itself. □. 

We denote by /i the unique U(n)-invariant measure on X„. The Jacobian of 
the transformation z ^ z^^l is given by the following formula 

/i(z[9l) = det($ + z^)-("+iV(-z) (2.2) 
A proof is the same as above (Lemma 1.3). 

2.3. Invariant kernels on U(n)/0(n). Now fix cr, r £ C. Consider the 
kernel La^r{z,u), where z, u G X„ the same kernel (|1.12f) restricted to X„. 
Obviously, the kernel L„^t{ztu) is U(7i)-invariant 

La.rih'^ zh, h^uh) = L„,r{z, u) 

We define the sesquilinear form on C°°(X„) by 

IaAFi,F2)= 1 1 L,,r{z,u)Fl{z)F^)d^l{z)d^i{u). (2.3) 

J JX„xX„ 
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Again, we consider the meromorphic continuation of I^t to the domain 
(a,r)eC2. 

As above, we define the linear operators 

Pr,a 1^ f{z) = det($+2*){-("+l)/2-^l|-(H+l)/2-r}_ 

As above, these operators preserve the form 1^^^- 

Theorem 2.2 Let n> 1. Let 2a, 2r ^ Z. The Hermitian form Ia,T is definite 
iff 

[-2T-n-l] = [2f7]. 

Proof is given below in Section 3. 

It is convenient to define new parameters s, thy 

a = -{n+l)/4 + s, T= -{n + l)/A + t. 

The conditions of positivity are 

a) For an even n, 

s, t e [i/2 - 1/4, j/2 + 1/4] for some j e Z. 

b) For an odd n 

s e b72, j/2 +1/4], te [j/2 - l/4, j/2] for some j G Z. 
For s = t the form I^^r defines the L^-inner product. 
B. Spaces U(2n)/Sp(n). 

2.4. The space U(2n)/Sp(n). Now we consider the space y„ of 2n x 2n 
matrices z satisfying the conditions 

zz* = 1, z = -z*. 

An example of z satisfying these conditions is 

J = ( ^ Y\ where 1 is the n x n unit matrix. (2.4) 



^-1 0^ 

The group U(2n) acts on this space Yn by the transformations 

h : Zi-* h^zh. 
The stabilizer of the point J € y„ consists of matrices satisfying 
hh* = 1, h*Jh = J. 
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These equations give one of the standard reaUzations of the compact symplectic 
group Sp(n). It is easy to show that y„ is a homogeneous space 

y„ = U(2n)/Sp(n). 

Using the transformations z ^ h^zh, where h £ Sp(n), we can reduce each 
element of y„ to the form 



= ( Q 1 , where A 



/e*'^i/2 ... \ 
e'^=/2 ... 



V ... e^-^-^V 



(2.5) 



The numbers are uniquely defined up to permutations and transformations (fij i— > 

2.5. Overgroup. We realize the group S0*(4n) as a group of complex 
(2n + 2n) x (2n + 2n) matrices g preserving the Hermitian form in C^" ® C^" 

with the matrix ^ and the symmetric bilinear form B having the matrix 

1 
-1 



^0 -Ij 
. In other words, 



1 \ / 1 \ t fo l\ /O 1 



-IJ ^ \0 -IJ ' " \1 OJ " \1 

These two identities also imply that the matrix g has the following block struc- 
ture 

9 - $ 

The group SO* (4n) acts on y„ by the linear fractional transformations 

g: z>-^z^^^ :={^-z^)-\^ + z'^). (2.6) 
Lemma 2.3 Let z &^n, 9 & S0*(4n). Then z^b] g y„. . 

Proof is the same as above. For a matrix ^ G y„, we consider its graph 
Vz C C^" © C^". As it was shown in 1.3, the subspacc 14 is maximal isotropic 
with respect to the Hermitian form H. The condition z = — z* means that 
is maximal isotropic with respect to the symmetric form B. 

Thus, we obtain the identification 

{Grassmannian of n-dimensional subspaces in C^" ® C^" 1 
isotropic with respect to the both forms H and B J 

{The space y„ = U(2n)/Sp(n) of 2n x 2n matrices! 
z satisfying z = —z*, z*z = ij 

Now the statement became obvious. 
2.6. Jacobian. 



16 



Lemma 2.4 For the linear-fractional transformations 1^2.61) . we have 

^(zM) = I dct($ - zl^)r('^"-2)^(^)^ 

Proof. The both expressions, 



:= det($ - z*); 1^2(9, z)-~ 



^i{z) 



(2.7) 



(2.8) 



satisfy the cocycle identity (|1.18|l mentioned above. 

Recall some standard facts about the solutions of the cocycle identity, see 
|85| . 13.2. Let G/R be a homogeneous space, let u be a point fixed by the 
subgroup R. Let v he a. solution of the cocycle equation on G/R. Obviously, 
for ri, r2 e M, 

v{rir2,u) = v{ri,u)v{r2,u). 

i.e., the function r i-^ v{r^u) is a character of R. Moreover, this correspondence 
between solutions of H1.18|l and characters of i? is a bijection. 

In our case, G = S0*(4n) and i? is a maximal parabolic in G, the reductive 
part of R is the group GL(ri,H). Each character of GL(n,]HI) has the form 
r 1-^- det(7')^, and hence our question is reduced to an evaluation of s. 

Since we must know the exponent s, it is sufficient to find 12. 8|) only for 
z — J and some appropriate g lying in the stabilizer of J. 

We choose g = r{t) being the block [n + n + n + n) x (n + ?i + n + n)-matrix 



r{t) 



/cosht sinhtX 

cosht — sinht 

— sinh t cosh t 

ysinht cosh^y 

A direct calculation shows that — J. Also 



det($- J*)"i =e- 
Applying the formula (|1.7|) , we obtain that the differential of z 



A 



(2.9) 

z[''(*)l at J is 
(2.10) 



Since dimy„ = dimU(2n) — dimSp(r7,) ~ 2n^ — n, the determinant of the 
transformation (|2.1Q(I is exp(4n(2n— l)i). Comparing with H2.9|l . we obtain the 
exponent in H2.7|l . □ 

2.7. Unitary representations of S0*(2n). We define the canonical in- 
variant kernel on y „ as the restriction of the kernel 

L^^riz,u) = det(l - zu*)^'"ll^> 

to ^n- We also define the Hermitian form on C°°(y„)) by 



L„^r{z, u)Fi{z)F2{u) dn{z) d^{u). 
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Theorem 2.5 Let n > 1. Let u, t ^ Z. The form L„^t is sign definite iff 

[-t] = [cr + 2n- 1]. 

The proof is contained in the next section. 

We define the representations pcr^r of S0*(4n) in C°°(y„) by 

p^.^ (^^- 1^ F{z) = F(($-2l^)-\*+zl)) det($-zl^)^-"+i/2-'^ll-"+i/2-^}. 

These operators preserve the form /o-.t- Under the conditions of Theorem 2.5, 
our representations are unitary. 

It is also convenient to introduce new parameters t, s by 

o- = -n+l/2 + s, r=-n+l/2 + t. 

Then the condition of positivity is 

\s - j\ < 1/2, \t - j\ < 1/2, for some j G Z. 

3. Application of the Kadell integral 

3.1. Jack polynomials. Preliminaries, for details see Macdonald, |37| . 
VI. 4, VI. 10, and further references in this book. We consider n-dimensional 
torus T" as a subset in C" consisting of points 

(xi , . . . , Xt^) , l-^ll — "^n I — 1- 

It is also convenient to write 

Xl — 6 , . . . , Xji — 6 . 

We denote by Sym„ the space of all polynomials in xi,. . . ,x„ symmetric with 
respect to permutations of xj . We denote by LSym„ the space of all symmetric 
Laurent polynomials in xf'^,. . . , x^^. 

Fix >f > 0. Consider the inner products in the spaces Sym„ and LSym„ 
given by 

{f,9)^= f{x)W) n \^k-xi\^^l[dcpk. (3.1) 

•^T" l^fc<i^n fc=l 

The Jack polynomials are orthogonal polynomials with respect to this scalar 
product. In a multivariate case, the Gramm-Schmidt orthogonalization of poly- 
nomials is not a canonical operation, and hence we must define them more 
carefully. 

Let A be a collection of integers (a Young diagram) 

A: Ai ^ A2 ^ ... ^ A„ ^ 0. (3.2) 
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Denote 

|A|=^A,. 

Let |A| = \fi\. We say A ^ /i iff 

Ai + • • • + Aj ^ fii + ■ ■ ■ + for all j. 

We define the monomial symmetric function m\ as 

where ... is the sum of all pairwise distinct monomials ■ ■ ■ 2^^('„)j where a 
is a permutation. 

The Jack polynomials Pxix) G Sym„ are defined by two conditions: 

a) Px are orthogonal with respect to the scalar product H3.1|l 

b) Px = mx+ ^ u^mf_t (3.3) 

Uniqueness of the Jack polynomials is obvious but existence is a theorem. 
The Jack polynomials satisfy the following identity 

-PAi + l,...,A„+l(a;) = Xi . . .XnPxi....,\„i^)- 

This allows to define Laurent Jack polynomials Px G LSym„ for 

A: Ai > A2 ^ ... ^ A„, AjGZ (3.4) 
(we permit negative Xj) by 

-Pa,...,A„ (x) ^ {Xi... Xny"^Pxi+Tn,....\„+m{x), 

where m is sufficiently large (A„ + m ^ 0). 

This system of polynomials forms an orthogonal basis in LSym„ with respect 
to the scalar product H3.1|l . 

3.2. Radial functions. Consider one of our symmetric spaces 

i^/iJ ^ U(n) X U(n)/U(n), U(n)/0(n), U(2n)/Sp(n). (3.5) 

Consider an _ff-invariant function F on G/H. Obviously, this function can 
be considered as a symmetric function depending in the invariants (^i,. . . , ipn 
defined respectively in (|2.1() . H2.5|l . 

Consider the map 

z"-^ {'fil,---,fn) (3.6) 

that takes a matrix z to its collection of invariants. Since ipj are defined up to 
a permutation, we must assume 

^ ipi ip2 ■ ■ ■ ^ fn < 2tt. 

The following variant of the Weyl integration formula holds. 
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Lemma 3.1 The push-forward}^ of the K -invariant measure /i under the map 
1^3. 6]) is given by the formula 

n 

const(>f)- Y[ le"^" -e"^'\^''Y[dipj, (3.7) 

l^fc<isjn j=l 

where 

— K = 1/2 /or U(n)/0(n) 

— K=\for U(n) X U(n) /U(n) 

— >f = 2 /or U(2n)/Sp(n) 

See, for instance, |29| . X.l, various calculations of this type are contained in 

Consider the space L?'{K/H)-^ consisting of 77- invariant L^-functions. To 
each function F G L^{K/H)^ , we consider the corresponding function / in 
variables ipj. By Lemma 3.1, we have 

/ i^i(z)7M^rf/i(z) = const / fi{v)h¥) n -e'^'\^'^f\d^, 

This is precisely the inner product (|3.1|l . 

3.3. Spherical functions. Preliminaries. Consider a symmetric space 

X/i7 = U(n) X U(n)/U(n), U(n)/0(n), U(2n)/Sp(n) 

The space L'^{K/H) is a multiplicity-free direct sum of (finite-dimensional) K- 
modules V„, 

L^{K/H) ~ (3.8) 

An irreducible if-module participates in this direct sum iff Vu contains 
a nonzero _ff-invariant vector v^}'^. By Gelfand's theorem (see |3U| ) . an H- 
invariant vector v^, G is unique up to a scalar factor. 

The spherical function of a module V^, is function on K defined by 

S,u{k) ■■= {k ■ Vu,v^)v^ (3.9) 

where v^, is normalized by the condition ||uy|| = 1. Obviously, for /ii, h2 E H 
we have 

Uhikh2)^Uk) 

This allows to consider the function as a function on our symmetric space 
G/H or as a function on the double coset space H \ G/H. 

In expansion (|3.8|) . we have G Vi,. Hence, the functions form an 
orthogonal basis in {K / H)^ . 

^''Let A be a space with a measure a, and h : A ^ B is a measurable map. The push-forward 
of the measure a is the measure (3 on B defined by P{S) = a{h~^ (S)), where S G B. 

^■'This statement is a variant of the Frobenius reciprocity, see 1351 . 13.1, 13.5. For description 
of modules satisfying this property, see, for instance, )30l . 
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The following fact is well-known, see, for instance, |S7], Chapter 7. 
H -spherical functions on K/H are the Jack polynomials (modulo some nor- 
malizing scalar factors.) the parameter >c was determined in Lemma 3.1. 

3.4. Reduction of our problem. Thus we intend to investigate a posi- 
tivity of the inner product 



{Fl,F2)a.r = / / i.,r(2, u)Fi{z)F2{u) d/i(z) 

J JK/HxK/H 

where L^r = det(l — zu*)^''^^'^^ is the distribution defined above. 

Let A be a collection (|3.4|) . P\{x; x) be the corresponding Jack polynomial, 
and V\ C G°°{K/H) be the if-invariant subspace containing P\{x,>c). 

Lemma 3.2 For F e C°°{K/H) consider its expansion F — '^x^^' where 
FxEVx. Then 

(F, G),,, V cx{a, t) f Fx{z)GK^dfi{z) 

X J K/H 

where cx{(J,t) are some constants. 

Proof. Consider the integral operator 

Uc,.tF{z)^ / L„,r{z,u)F{u)d^i{u) 

J K/H 

Since the if-invariance of the kernel La.T^ the operator Ua^r is an intertwining 
operator for K. Since the action of K in C°° is multiplicity-free, the restriction 
of Ua^r to Vx is a scalar operator (see 8.3), i.e., 



Ua,rFx{z)^ L„^riz,u)Fxiu)dfi{u) ^cx{a,T)Fx{z) (3.10) 

J K/H 

and this implies the required statement. □ 
Next, we must evaluate the constants ca(ct, t). For this purpose, we substi- 
tute Fa = Pa and 

— z = 1 in the case K/H = U(7;.), 

— z = 1 in the case K/H = U(n)/0(n), 

— z = J (see (E31) in the case K/H = U(2n)/Sp(n). 
In all the cases we obtain the integral 

Lx{>c;cr,T) = 

P n n 

= [](1 -e*'^'=){'^ll^>PA(e''^S...,e*'^";>f) J| \e"f"' - e"f"\^'' ^dipj 

k=l l^k<l^n j=l 

(3.11) 

where x is the same as in Lemma 3.1. 
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In the first two cases, we obtain this integral immediately, for the case 
U(2n)/Sp(n) we must write det(l - Jfl), where J is (|2.4I) . and SI is (|2.5() . The 
matrix det(l — Jft) is diagonal, and 



det(l -,m) = Y[{l~- e''/''=/2){'^ll^}(l _ e-»¥'./2){<T||r} = - e»'P.){<T||r} 

fe=i fc=i 

3.5. The Kadell integral. Let k he a, positive integer. Let A satisfies 
The Kadell integral (see El, [S3, VLIO, Example 7) is given by 



k=l l^k<l^n j=l 

= ax{K;r,s)vx{K) (3.12) 



3C 

n 

[0,1] 



where 

r(A,-A, + >.(^-fc)) ^ ^ 

aA(>f;r,s =11 — — ■ ■ — — (3.14 

r[Xj + r + s + x{2n - J + 1)) 

3.6. Transformation of the Kadell integral. 

Proposition 3.3 Let Re(o' + r) > —1, Rex > 0, and A satisfies \3.4\l - Then 
the integral !i\(>c\(T,T) given by iS.llp equals 

Lx{K;a,T) = (27r)"n!wA(><)x 

A {-l)^^r{a + r + l + K{n-j)) 

r(-A, +T+l + >c{j^ 1))T{\, +a + l + K{n - j)) ^ ' ^ 



X 



where vx{x) is the same as above fS.lS}) . 

Proof. First, let x he a positive integer. Assuming Xk = e*''''', 

|g^c,. _ g^^, |2 ^ |^^_ _ = {xk - xO(a;^' - xr') = -(x,- - xO^x^^a;;-' 

Next, transform the expression 13.11|l to a contour integral 

P n n 

J P,{x;>c)l[x-^-''^^''-''>-\l-Xkr+- n {^k~xif''Y[dx 



X 

|a;i| = --- = |a;„ 1 = 1 



3 

k=l l^A;<i^n i=l 
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Now the integrand is holomorphic in the polydisc \zk\ < 1 outside the cut 
xi G [0, 1], . . . , x„ G [0, 1]. We define the branches of the factors by 



= 1, x'- 

x=0 



The domain of convergence of this integral is 

Re(cr + r)>-l (3.16) 

Then we deform each one-dimensional contour Xj = e^^^ to the contour 
around the segment Xj = [0,1] lying in an infinitely thin strip |Ima;j| < e. 
Thus our expression converts to 

(-_-^yi(n-i)V2(_2sin7rT)" / (the same integrand) 

^[0,1]" 

The domain of convergence is reduced to 

Re(cr + r)>-l, ReT<-><(n-l) (3.17) 
Applying the Kadell formula, we obtain 

Six{>c\<7,T) = 2"n!(-sin(T7r))"(-l)"("-i)^/2^A(>?)x 

r(A,- - x{3 - l))r(a + r + l + K{n - j)) 

r(A,+a + l + x(n-,)) ('-^'^ 

Then we write 



— sin(7rT)r(Aj — r — >f(j — 1)) = 

= (_1)^.— — 0-1) sin(A, - r - - l))^)r(A, - r - x(j - 1)) 

(again, we use the condition >^ G Z). Applying the reflection formula 

sin(7rz)r(z) = 7r/r(l - z) 

we obtain H3.15|l . 

Thus, we have the required identity (|3.15|l under the condition (|3.17|) . But 
the both sides of H3.15|l are analytic in the domain (|3.1t)|) . Hence the identity is 
valid in this domain. 

Thus the statement is proved for an integer positive x. 

Next, we intend to apply the following Carlson theorem, see 2.8.1. 

— Let f{z) he an analytical function in the domain Re z > and f{z) = 
0(e"l^l) with some a < n. If f{n) = for n = 1, 2,. . . , then f{z) is identically 
zero. 
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Fix real positive ct, t. The identity H3.11|l is valid for positive integer >c. We 
intend to show that it is valid for Rc>f > 0. First, we slightly change the factors 
of the integrand 

Accordingly we multiply the right-hand side of (|XTT|) by 2-"("-i)^-"(^+^) . Now 
the expression IljLi ' ' ' ni<j<fc<n ■ ■ ■ integrand is < 1. 

For coefficients of the Jack polynomials in formula there exists a 

semi-explicit expression that is rational in the variable k, see |37| . VI. 10, page 
379. The poles of these expressions are some non-positive rational numbers. 
Hence, for a fixed A, the expression P\{x; x) admits a holomorphic continuation 
to the domain Re>^ > 0, and moreover Px{x; has a at most a polynomial 
growth in >c. 

Thus our integrand has (at most) a polynomial growth in x. Hence the same 
statement is valid for the integral. 

Now we estimate the growth of the right-hand side of l3.15l First, our formula 
is valid for Ai = • • • = A„ = 0, (in this case, we obtain the Cauchy-type form 
of the Selberg integral, see jlj; see also (10.38), for A = 0). The integrand 
now is less than 1, and-hence the right-hand side also is bounded. 

Consider the ratio of right-hand sides 

Lxix; a,T)/Zo{K; a,T) 

This ratio is the product of expressions having the form r(/i + lx)/T{iy + Ik). 
This product has a polynomial growth in since 

?7Jt4 - ^ ^ 

r(6 + z) 

(see, for instance PP, 1.4.3; the asymptotic is valid in the domain | argz| < tt— e). 

Now we have two functions having a polynomial growth as \z\ — > oo, they 
coincide in positive integers and by the Carlson theorem they coincide in the 
half-plane. 

It remains to omit the condition A„ ^ 0. But the transformation 

Aji-^Aj — 1, TH^r — 1, a ^ G + 1 

does not change the both sides of our integral, and hence our formula can be 
used for general A satisfying (|3.4(l 

3.7. Positivity. We must analyze, when H3.15|l has constant signs for all 
A satisfying (|3.4I) . The factor vx given by H3.13|l is positive. Hence wc must 
analyze positivity of the factor H ■ • ■ l|3.15|l . Equivalently we must trace the 
constancy of the sign of the factor 



n 



r(A, - r - k{j - 1)) 
L r(Aj + cr + 1 + x(n - i)) 
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in (EHJl. 

The case >c = 2. We must analyze positivity of the expression 

fr r(A, - r - 2, + 2) 

The function r(a;) changes its sign at the points a; = 0, —1, —2, .... Hence the 
condition 

[-T-2.7 + 2] = [a + l + 2(n-j)] <^ [-r] = [a + 2n - 1] (3.20) 

is sufficient for positivity. 

Let us show that this condition is necessary. Assume that all the Xj are 
negative integers having large absolute values, and assume that — Xj also are 
large. Then the sign of (|3.19|) depends only on parities of Xj. Also, it is clear, 
that all the factors in (|3.19() have to be positive. But this implies (|3.20() . 

The case >f = 1 is similar. We must examine the positivity of 



n 



r(A,-T-j + i) 



7=1 ^(^J + a + n - J + 1) 

The case yt = 1/2. We must examine the positivity of 

^ r(A,--T-j72 + l/2) 
lir(A,+a + l + (n-j)/2) 

Considering even j-s, we obtain 

\-T + 1/2] = [a + 1 + 71/2] 

Considering odd j-s, we obtain 

[-T + 1/2 + 1/2] = [a + 1 + 71/2 + 1/2] 

This implies Theorem 2.5. 

4. Remarks on general case 

Here we discuss Stein-Sahi representations of arbitrary classical groups. 

4.1. Flat matrix spaces. We consider the following 10 types of spaces M 
of n X 71 matrices; in each case we write groups G Z) K Z) H ^ the sence of this 
notation will be explained below; if ti have to be odd, then we write n — 2k 

— 71 X 71 matrices over C; 

GL(2n,C) D U(27i) D U(n) x U(?i); 

— 71 X 71 Hermitian matrices over C; 

U(73,n) D U(7i) X U(7i) D U(n) 
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— n X n symmetric matrices over C; 

Sp(27i,C) D Sp(n) D U(n) 

— 2k X 2k skew-symmetric matrices over C; 

S0(2A:,C) D S0(2fc) D U(fc). 

— n X n matrices over R; 

GL(2n,M) D 0(2n) D 0(n) x 0(n) 

— n X n symmetric matrices over M; 

Sp(2n,R) D U(n) D 0(n) 

— 2k X 2k skew-symmetric matrices over R 

SO(2fc,2fc) D 0(2fc) X 0(2fc) D 0(2fc) 

— n X n matrices over H, 

GL(2n,H) D Sp(2n) D Sp(n) x Sp(n) 

— n X n anti-Hermitian matrices over M, 

Sp(n,ri) D Sp(n) x Sp(n) D Sp(n) 

— k X k Hermitian matrices over H; 

S0*(4fc) D U(2fc) D Sp(fc) 
The group G is isomorphic to the group of hnear-fractional transformations 

X ^ x^^^ = {a + xcy^{b + xd) 

preserving the symmetry condition. The Jacobian of such transformation is 



J(.g, x) — det \ {a + xc)\ det ^ 



a b 
c d 



where 

2 

h = - dimM, 
n 

n is size of the matrices, and (3 — ndimK in 3 cases then we consider ah 
the matrices without conditions of symmetry (i.e., G — GL(ri, K)); in all the 

remaining cases |det ^ ^ ^ ^' ^'^^^^^^ 

The group K is the maximal compact subgroup in G, and H C K is the 
stabilizer of 0. In all the cases, K/H is a compact symmetric space, and M is 
an open dense subset in K/ H 

Remark. In all the cases, there exists some Grassmannian type realisation 
of K/H in spirit of our Subsections 1.3, 2.2, 2.5, see a table in 

We define the Stein-Sahi inner product in C°°(M) by 



{F,,F2)= \Aei{x-y)\''F^{x)F2{y)dxdy (4.1) 

J JMxM 

The group G acts in the space of functions on M by formula 

'°(c ^) ^(a;) =^((a + a;c)-i(6 + xd))|det(a-f a;c)~''-^ (4.2) 
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these transformations preserve the Hermitian form 6. 

4.2. Existence of an island of positivity. First, consider the represen- 
tation of G in functions on M given by 

F{x) ^ /((a + a;c)-^(6 + xd)) det(a + xc)-''/'^ (4.3) 

It is a representation of a degenerated unitary principal series. 

Consider the maximal parabolic subgroup P C G consisting of matrices hav- 
ing the form . This subgroup acts on functions by afhne transformations 

F{x) y-^ F{a-\b + xd)) det(a)" det((^)^ (4.4) 
where u, d are some explicit real numbers (non-interesting for us). 

Lemma 4.1 If G ^ Sp(2n,M), U(n, n), S0*(4n), then the representation \4-4}^ 
of P is irreducible. 

Proof. Consider the subgroup N C P consisting of matrices ^ , it 

acts by shifts f{x) i-^ f{x + b). 

Also consider the subgroup L C P consisting of matrices ^ , it acts by 

the transformations f{x) >— > f(axd^^). 
After the Fourier transform 



F{x) ^ / F(a:) exp(iRetra;^*) dx 
Jm 



the group N acts via multiplications by functions 

$(0 ^ *(0exp(iRetrf6*) (4.5) 

Assume that the representation (|4.4|l is reducible. Then there exists an inter- 
twining projector H. In the Fourier model, it commutes with all the operators 
(|4.5|l and hence 11 is a multiplication by a function x taking values and 1. 
But this operator also must commute with the subgroup L, which acts by the 
transformations having the form 

$(C) ^{A^D-^)\detA\"'\detD\''' 

But in all the cases, the group of transformations ^ ^ AS^D^^ has an open orbit 
on M and hence x is a constant. □ 

Corollary 4.2 If G ^ U(n,n), Sp(2n,R), S0*(4n), then the representation 
{j--^ is irreducible. 

Since (|4.3I) is irreducible, it admits a unique invariant Hermitian form and 
this form is the L^-inner product. Representations H4.2|l lying near H4.3|l admit 
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the invariant Hermitian form H4.1|l . and since this Hermitian form is a minor 
deformation of L^-inner product, this Hermitian form also is positive definite. 
It is not difficult to justify these heuristic arguments in this case (there are 
abstract theorems of this type, see [3^], Theorems 6, 8; unfortunately they do 
not cover our cases). 

Description of the poles of the distribution |det(x)|^ that appear in (|4.1|l 
is a relatively standard problem, see Sato, Shintani |Snj^^. Knowledge of these 
poles allows to find precisely the interval of positivity. We omit details. 

4.3. Other possible ways of proof. Existence of Stein-Sahi series is 
a relatively simple fact, and apparently it can be proved uniformly in various 
ways. 

One possibility is to use the work of Branson, Olafsson, 0rsted, 
Another way is to use our method of Sections 1-3, but a necessary identity 
for multivariate Jacobi polynomials (see |28p in my knowledge is not known 
yet. But for the groups G — 0{2n,2n), Sp(n,n), GL(2n,C) this way can easily 
be realized (the first two cases are discussed in ,62 ) due existence of explicit 
elementary formulas (Weyl's and Berezin-Karpelivich's) for spherical functions. 

5. One problem of non-L^ harmonic analysis. 
A. Construction of kernels 

5.1. Uniform realizations of symmetric spaces. Consider the following 
classical groups 



Q-GL(n,C), GL(n,R), GL(n,H), 0(p,g), U(p,g), Sp(p,g), 0(n,C), 



Remark. Our list contains GL(n, K) and not SL(n, K), also U(p, q) and not 
SU(p, g). Modulo this remark, our list contains all the classical groups. 

Consider an afhne symmetric space having the form Q/Y, where 1^ is a 
symmetric subgroup in Q (according the Berger classification, there are 54 series 
of classical affine symmetric spaces). It turns out to be (see [HSl) that the space 
Q/S admits a realization as a set, whose points are pairs of complementary 
subspaces {V, W) in some linear space K™, these subspaces satisfy some simple 
conditions (as an isotropy with respect to some form, orthogonality, or existing 
of a given involution transposing two subspaces). 

Example 1. The unitary group U(n). Consider the space C"©C" equipped 
with the Hermitian form H having the matrix [ ^ ^ j as above (Subsection 



^^For more information on determinant distributions (explicit expressions at poles, supports, 
etc.) see works Rais 73|, Ricci, Stein [7^, Muro 



Sp(2n,C), Sp(2n,M), S0*(2n) (5.1) 




Consider 



the set U, whose points are pairs of subspaces (V, W) such that 



0*. C"eC" ^v®w 
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1*. V and W are iJ-isotropic. 
2*. W ^JV 

Now let z : C" — > C" be an unitary operator. Let ^ be a graph of z, and 
W be the graph of (— z). Then V, W satisfy to the conditions 1-3. It can be 
readily checked that all the pairs satisfying 1-3 have this form. 

Thusl[:iiU(n).i6 

Example 2. The space U(n, n)/GL(n, C). Consider the same space C"0C" 
equipped with the same Hermitian form H . Consider the set IK, whose points 
are pairs V, W of subspaces satisfying the conditions 0* — 1* from the previous 
example. Obviously, IK ~ U(n, n)/GL(n, C). 

Example 3. The spaces U(n, n)/U(p, g) x U(n - p,n — q). Consider the 
same space C" © C" with the same form H. Consider the set 9, whose points 
are pairs of subspaces (V, W) satisfying the conditions 0* and 

— dim V = a 

— W is the orthocomplement of V with respect to H. 

Open orbits of the group U(n, n) on S are enumerated by the inertia indices 
of H on V. Thus" 

S — Up+g=aU(n, n)/U(p, q) x U(n -p,n- q). 

It appears that enumerating carefully all the possible conditions of this type, 
we obtain precisely the classical part of the Berger list.^^ 

5.2. Overgroup of the symmetric space. In Examples 1 and 3, the 
subspace W is determined by the subspace y, and hence we have an embedding 
of the symmetric space to some Grassmannian G/P^^; the same phenomenon 
holds for 44 series of symmetric spaces. 

In Example 2, the subspaces V, W are 'independent', and we obtain an open 
embedding of a symmetric space to a product of two Grassmannians G/Pi x 
G/P2^° (and this happens for remaining 10 series). 

In particular, we obtain that for each classical symmetric space Q/Y there 
exists^^ larger group G acting locally on Q/Y . 

Remark. It seems that this (trivial) observation was firstly claimed in |53| . 
But overgroups of symmetric spaces were discussed by different reasons by many 
authors, in particular Hua Loo Keng, Nagano, Takeuchi, Goncharov, Gindikin, 
Kaneyuki. Nagano gave a complete analysis of such overgroups in Rieman- 
nian case (including exceptional cases). Makarevich |2H1 in 1973 classified open 

^'^Of course, the subspace W in our construction seems artificial and can be forgotten (as 
it was done above in Sections 1-2). But it take part in a general construction of the kernel 
below) . 

^^Again, it seems that W is an artificial element of the construction (since W is determined 
by y), but the transversality condition 0* involves W . 

-"^^We must consider all the fields K = R, C, H, all possible natural forms and all possible 
involutions. For a table, see 1531 . 

^^In both examples G = U(n,n) 

^"in our example G = U(n, n) 

^-'^ Emphasis that Q is contained in the list 15. li . For instance, U(n,n) acts on the space 
U(n); but there is no group G D SU(n) X SU(n) acting on the hypersurface SU(n) C U(7i) 
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orbits of reductive groups in Grasmannans. In fact, the right-hand side of long 
Makarevich's tables includes the whole classical part of Berger's list, and this 
implies our observation. As far as I know, nobody actually compared these 
Bergers's and Makarevich's lists during 25 years. 

5.3. Double ratio of 4 subspaces. Consider a linear space K''+''. Let Vi, 
Wi, V2, W2 be four subspaces in and 

dim Vi = dim V2 = p, dim Wi = dim W2 = q 

For a quadruple being in general position, we define a canonical operator (the 
Hua Loo Keng double ratio) X) : Vi ^ Vi by the following rule. Since IK''+'' = 
Vi ®Wi, the subspace V2 is a graph of an operator A : Vi ^ Wi, and _B is a 
graph of an operator B : W\ V\. We assume 

D(Fi,W^i,V2,T^2) = BA 

The operator is canonically defined, in particular, its eigenvalues are invari- 
ants of a quadruple of subspaces. 

By Subsection 5.1, the double ratio is well defined in each classical symmetric 
space. 

5.4. A problem. Different formulations. The author think, that for 
the following problems of the harmonic analysis, the explicit Planchcrcl formula 
can be obtained. The arguments for support of this point of view are discussed 
below. 

We use the notation of Section 4. 

a) Tensor products. For a given group G from the list (|0.7H ine2'l. decompose 
a tensor product of two Stein-Sahi representations. 

b) Restrictions. Let Q/Y be a symmetric space. Assume G that its over- 
group has the Stein-Sahi representations. Assume that the local action of G 
on Q/Y is locally isomorphic to the action G on the matrix space, see 4.1. 
Decompose the restriction of a Stein-Sahi representation to G. 

Remark. The problem about tensor products is a problem of the restriction 
of a representation p ® p' oi the group G x G to the diagonal subgroup G. 

c) Natural kernels on symmetric spaces. Let Q /Y he the same as above. Let 
restrict the Stein kernel L(-, ■) to Q/Y . Consider the inner product in C'^{Q/Y) 
defined by the kernel L (see (|4.1|) ') and the unitary representation of Q in this 
space. To find the Plancherel formula for this representation. 

Remark. This question slightly differs from the previous one in the case, 
then Q has several open orbits Q/Yj on the Grassmannian, see Example 3 in 
5.1. 

d) More general kernels on symmetric spaces. Consider a classical symmet- 
ric space Q/Y realized as in 5.1, i.e., points of Q/Y are pairs r — \V,W] of 

^^Only few exceptional symmetric spaces admit such overgroup; see IIUI . 
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complementary subspaces. We define (see Addendum to ^7\) the kernel K{-, •) 
on Q/Y by 



K{[Vi,Wi],[V2,W2]) = dot 



^{Vl,WuV2,W2) 



e 



1- 3(1/1,1^1, 1/2, W^2) 



and the inner product on C°°{Q/Y) given by 




Q/yxQ/y 



K{ri,r2)fi{ri)f2{r2) dfi{ri) d^i{r2) 



where is the Q-invariant measure on Q/Y. 

If the overgroup G of Q/Y admits Stein-Sahi representations^'^, then our 
construction is equivalent to the previous one. 

Otherwise, we obtain a problem of positivity of the inner product. There 
are spaces Q/Y, for which islands of positivity are absent (|nni)i and the cases, 
when such islands exist (02; ESI)- Generally, the problem is open. 

Again, our question is to find the Plancherel measure. 

I do not know has this question sense if there is no positivity, see a discussion 
in Addendum to ^!7\ . 

e) More tensor products. Our Stein representations of G are induced from 
some parabolic P. We ask about the tensor products of arbitrary two represen- 
tations induced from this parabolic. 

B. Discussion 

Below we discuss the variant b) of the problem, i.e., the restriction of a Stein 
representation to a symmetric subgroup. 

5.5. Approximation of on symmetric spaces. First, for all the 
groups G, Stein-Sahi series includes some representation (or representations) of 
a principal series (in the notations of 1.8, 2.3, 2.7, this corresponds to s = i, in 
the notation of section 4, this corresponds to 6* = —h/2. 

In this case, our problem is equivalent to decomposition of L'^{Q/Y). 
Hence our representations are some kind of deformation of L'^{Q/Y) 

5.6. Modern picture of on a pseudo-Riemannian symmetric 
space. The problem of decomposition of on an arbitrary pseudo-Riemannian 
symmetric space was considered as important after the Flensted-Jensen's work 
on discrete series ^^li 1980. 

After large efforts, the problem is not completely solved up to now. 
First, in the following partial cases, the explicit solution is known. 

— Riemannian symmetric spaces G/K (Gelfand-Naimark, |22j for complex 
classical groups, Gindikin-Karpelevich, j^Hlj for general case). 

— Semisimple Lie groups (Gelfand-Naimark, for complex classical groups, 
Harish-Chandara, 27 , for general case 

^''To avoid ambiguity, we also must require that the local action of G on Q/y is equivalent to 
the action on the matrix spaces (or Grassmannians) from 4.1. This slip in speech is important 
only for G = GL(n,K). 
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— Rank 1 spaces, Molchanov (|321, SHI)- 

— The spaces of the form Gc/Gr, where Gc is a complex group, and Gr is 
a real form (Harinck, jJS], |26)). 

Second, there was a long story of search of the general Planchcrel formula. 
Author do not intend discuss it and fix modern situation. 

There are two groups of authors, van den Ban - SchlichtkruU [5], 0] and 
Delorme - Carmona |12) proposed variants of a Plancherel formula (earlier a 
Plancherel formula was announced by Oshima, but proof was not published). 
Proofs are very heavy and occupy large collections of papers; also these formulae 
are nonexplicit and the c-function is not evaluated. 

Oshima ^ recently announced (with sketches of proofs) an explicit formula 
for c-function for the " most continuous part of spectrum" . Apparently, Oshima's 
formula together with van den Ban-SchlichtkruU's residue calculus allow to 
receive a final solution. 

A more important problem is to make this branch of analysis available to a 
wider mathematical community. 

5.7. Degenerated cases of our problem. Consider the representations 
Pcr.r of U(n,n) defined in 1.6. Consider a tensor product po-,T ^ Pa',T'- By 
definition, this representation acts in a space of functions on U(n) x U(n). The 
diagonal group U(n, n) C U(n, n) x U(ri, n) has an open orbit U(n, n)/GL(ri, C) 
on U(n, nY^. 

Thus, U(n,n) acts in the space of functions on U(n, n)/GL(n, C). 
Representations pcr,o are highest weight representations of U(ri, n), and po.r 
are lowest weight representations. The problems of decomposition 

are degenerated cases of our problem. 

In the first case, we have purely discrete spectrum, and problem has com- 
binatorial nature (see, for instance j33j), in the second case we obtain 'Berezin 
representations' discussed in Introduction. 

Hence our problem admits a degeneration into 2 important problems. 

Restriction of a highest weight representation po-.o of G to a symmetric sub- 
group Q leads to a similar alternative, i.e., we have purely discrete spectrum^^ 
or Berezin representation^^ 

But generally such restriction problems leads to harmonic analysis on some 
class of spaces that is more general than symmetric spaces. 

^^Indeed, a point of U(n) can be considered as a pair of maximal isotropic subspaces in 
C" ® C", see Example 2 from 5.1. A stabilizer of a pair of maxomal isotropic subspaces is 
GL(n,C). 

^^An example is G = U(n, n), Q = U(p, q) X U(ra — p,n — q), the corresponding symmetric 
spaces are described in Example 3 of 5.1. 

^^An example is U(n, n) D 0(n, n). This restriction problem can be interpreted as a problem 
of analysis on the symmetric space 0(n, n)/0(n, C). 

■^'^Some counterexamples are the restriction problems U(p, g) D 0{p,q) for p ^ q and 
U(n,n) D Sp(2n,R) for odd n. 
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5.8. Rank 1 cases. 

a) Tensor products of unitary representations SL(2, R) — SU(1, 1). For linear 
representations of SL(2, R), the spectra were determined by Pukanzsky ^72^ and 
the Plancherel formula by Molchanov [3^1, |4(Jj. 

Even this relatively simple problem is unexpectedly non-trivial and not per- 
fectly understood up to now^* '^'^ , see recent works |24) . 

Other rank one cases. There are many explicit calculations in the rank one 
case, that can be attributed to our subject, see 021-01], 0^, 01], ^1], our list 
do not pretend to be complete. 

5.9. Compact symmetric spaces. Let our space Q/Y is compact. For 
instance, let Q/Y is U(7i). Then we have the space of functions on U(n) with 
the inner product (1.15). The group U(n) x U(n) acts in this space by trans- 
formations F{z) I— > F{a~^zb). Search of the Plancherel formula is equivalent to 
expansion of Lo-,r in spherical functions (modulo some correction factors). 

Since we have obtained explicit expansions, we also know the Plancherel 
formula for the corresponding compact symmetric spaces. 

5.10. Oshima's argument. In [S7], Oshima shows that for two spaces 
Qi/Yi and Q2/Y2 having the same complexification, their c-functions satisfy the 
same difference equations. In particular, their ratio a priory is a trigonometric 
function. 

Apparently, this phenomenon survives in our case. Since in the compact 
case the Plancherel formula can be obtained, it is natural to believe that it can 
be obtained in a general case.^^ 
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